{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Tensorflow\n",
    "\n",
    "This notebook will familiarize you with the **basic concepts**\n",
    "of Tensorflow. Each of these concepts could be extended into\n",
    "its own notebook(s) but because we want to do some actual\n",
    "machine learning later on, we only briefly touch on each of\n",
    "the concepts.\n",
    "\n",
    "Table of Contents:\n",
    "\n",
    "- [ 1 The Graph](#1-The-Graph)\n",
    "- [ 2 The Session](#2-The-Session)\n",
    "- [ 3 The Shapes](#3-The-Shapes)\n",
    "- [ 4 Variables – bonus!](#4-Variables-%E2%80%93-bonus!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.4.1'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "# Always make sure you are using running the expected version.\n",
    "# There are considerable differences between versions...\n",
    "# We tested this with version 1.4.X\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 The Graph\n",
    "\n",
    "Most important concept with Tensorflow : There is a Graph to which\n",
    "tensors are attached. This graph is never specified explicitly but\n",
    "has important consequences for the tensors that are attached to it\n",
    "(e.g. you cannot connect two tensors that are in different graphs).\n",
    "\n",
    "The python variable \"tensor\" is simply a reference to the actual\n",
    "tensor in the Graph. More precisely, it is a reference to an operation\n",
    "that will produce a tensor (in the Tensorflow Graph, the nodes are\n",
    "actually operations and the tensors \"flow\" on the edges between\n",
    "the nodes...)\n",
    "\n",
    "Important note : There is a new simplification of the execution theme\n",
    "presented in this notebook :\n",
    "[Tensorflow Eager](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/eager/python/examples/notebooks/1_basics.ipynb)\n",
    "-- But since this feature is currently in alpha state and most code\n",
    "still uses the graph/session paradigm, we won't use the new execution\n",
    "style (anyways, if you can use the old style the new simplified style\n",
    "should be a welcome and straight forward simplification...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.framework.ops.Graph at 0x7f779928ad90>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# There is always a \"graph\" even if you haven't defined one.\n",
    "tf.get_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Store the default graph in a variable for exploration.\n",
    "graph = tf.get_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Ok let's try to get all \"operations\" that are currently defined in this\n",
    "# default graph.\n",
    "# Remember : Placing the caret at the end of the line and typing <tab> will\n",
    "# show an auto-completed list of methods...\n",
    "graph.get_operations() #graph.get_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "False\n",
      "False\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Let's create a separate graph:\n",
    "graph2 = tf.Graph()\n",
    "\n",
    "# Try to predict what these statements will output.\n",
    "print tf.get_default_graph() == graph\n",
    "print tf.get_default_graph() == graph2\n",
    "with graph2.as_default():\n",
    "    print tf.get_default_graph() == graph\n",
    "    print tf.get_default_graph() == graph2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"Const:0\", shape=(3,), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "# We define our first TENSOR. Fill in your favourite numbers\n",
    "# You can find documentation to this function here:\n",
    "# https://www.tensorflow.org/versions/master/api_docs/python/tf/constant\n",
    "\n",
    "# Try to change data type and shape of the tensor...\n",
    "\n",
    "favorite_numbers = tf.constant([13, 22, 83])\n",
    "\n",
    "print favorite_numbers\n",
    "\n",
    "# (Note that this only prints the \"properties\" of the tensor\n",
    "# and not its actual value -- more about this strange behavior\n",
    "# in the section \"The Session\".)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Operation 'Const' type=Const>]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Remember that graph that is always in the background? All the\n",
    "# tensors that you defined above have been duefully attached to the\n",
    "# graph by Tensorflow -- check this out:\n",
    "# (Also note how the operations are named by default)\n",
    "\n",
    "graph.get_operations()  # Show graph operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'Const:0'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Note that above are the OPERATIONS that are the nodes in the\n",
    "# graph (in our the case the \"Const\" operation creates a constant\n",
    "# tensor). The tensors themselves are the EDGES between the nodes,\n",
    "# and their name is usually the operation's name + \":0\".\n",
    "favorite_numbers.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's say we want to clean up our experimental mess...\n",
    "# Search on Tensorflow homepage for a command to \"reset\" the graph:\n",
    "# https://www.tensorflow.org/api_docs/\n",
    "\n",
    "### YOUR ACTION REQUIRED:\n",
    "# Find the right Tensorflow command to reset the graph.\n",
    "tf.reset_default_graph() #tf.\n",
    "tf.get_default_graph().get_operations()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Important note: \"resetting\" didn't clear our original graph but\n",
    "# rather replace it with a new graph:\n",
    "tf.get_default_graph() == graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Because we cannot define operations across graphs, we need to\n",
    "# redefine our favorite numbers in the context of the new\n",
    "# graph:\n",
    "\n",
    "favorite_numbers = tf.constant([13, 22, 83])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "favorite_products.shape= (3,)\n",
      "favorite_sum.shape= ()\n"
     ]
    }
   ],
   "source": [
    "# Now let's do some computations. Actually we don't really execute\n",
    "# any computation yet (see next section \"The Session\" for that), but\n",
    "# rather define how we intend to do computation later on...\n",
    "\n",
    "# We first multiply our favorite numbers with our favorite multiplier:\n",
    "favorite_multiplier = tf.constant(7)\n",
    "# Do you have an idea how to write below multiplication more succinctly?\n",
    "# Try it! (Hint: operator overloading)\n",
    "favorite_products = tf.multiply(favorite_multiplier, favorite_numbers)\n",
    "print 'favorite_products.shape=', favorite_products.shape\n",
    "\n",
    "# Now we want to add up all the favorite products to a single scalar\n",
    "# (0-dim tensor).\n",
    "# There is a Tensorflow function for this. It starts with \"reduce\"...\n",
    "# (Use <tab> auto-completion and/or tensorflow documentation)\n",
    "### YOUR ACTION REQUIRED:\n",
    "# Find the correct Tensorflow command to sum up the numbers.\n",
    "favorite_sum = tf.reduce_sum(favorite_products) #favorite_sum = tf.\n",
    "print 'favorite_sum.shape=', favorite_sum.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Because we really like our \"first\" favorite number we add this number\n",
    "# again to the sum:\n",
    "\n",
    "favorite_sum_enhanced = favorite_sum + favorite_numbers[0]\n",
    "# See how we used Python's overloaded \"+\" and \"[]\" operators?\n",
    "\n",
    "# You could also define the same computation using Tensorflow\n",
    "# functions only:\n",
    "# favorite_sum_enhanced = tf.add(favorite_sum, tf.slice(favorite_numbers, [0], [1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "within with:\n",
      "<tensorflow.python.framework.ops.Graph object at 0x7f77992bae90>\n",
      "Tensor(\"within_with:0\", shape=(3,), dtype=int32)\n",
      "[<tf.Operation 'within_with' type=Const>]\n",
      "\n",
      "outside with:\n",
      "<tensorflow.python.framework.ops.Graph object at 0x7f77992a3d10>\n",
      "Tensor(\"within_with:0\", shape=(3,), dtype=int32)\n",
      "[<tf.Operation 'Const' type=Const>, <tf.Operation 'Const_1' type=Const>, <tf.Operation 'Mul' type=Mul>, <tf.Operation 'Const_2' type=Const>, <tf.Operation 'Sum' type=Sum>, <tf.Operation 'strided_slice/stack' type=Const>, <tf.Operation 'strided_slice/stack_1' type=Const>, <tf.Operation 'strided_slice/stack_2' type=Const>, <tf.Operation 'strided_slice' type=StridedSlice>, <tf.Operation 'add' type=Add>]\n"
     ]
    }
   ],
   "source": [
    "# Of course, it's good practice to avoid a global invisible graph, and\n",
    "# you can use a Python \"with\" block to explicitly specify the graph for\n",
    "# a codeblock:\n",
    "with tf.Graph().as_default():\n",
    "    within_with = tf.constant([1, 2, 3], name='within_with')\n",
    "    print 'within with:'\n",
    "    print tf.get_default_graph()\n",
    "    print within_with\n",
    "    print tf.get_default_graph().get_operations()\n",
    "print '\\noutside with:'\n",
    "print tf.get_default_graph()\n",
    "print within_with\n",
    "print tf.get_default_graph().get_operations()\n",
    "\n",
    "# You can execute this cell multiple times without messing up any graph.\n",
    "# Note that you won't be able to connect the tensor to other tensors\n",
    "# because we didn't store a reference to the graph of the with statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting _derived/2_visualize_graph.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile _derived/2_visualize_graph.py\n",
    "# (Written into separate file for sharing between notebooks.)\n",
    "\n",
    "# Let's visualize our graph!\n",
    "# Tip: to make your graph more readable you can add a\n",
    "# name=\"...\" parameter to the individual Ops.\n",
    "\n",
    "# src: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/deepdream/deepdream.ipynb\n",
    "\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from IPython.display import clear_output, Image, display, HTML\n",
    "\n",
    "def strip_consts(graph_def, max_const_size=32):\n",
    "    \"\"\"Strip large constant values from graph_def.\"\"\"\n",
    "    strip_def = tf.GraphDef()\n",
    "    for n0 in graph_def.node:\n",
    "        n = strip_def.node.add() \n",
    "        n.MergeFrom(n0)\n",
    "        if n.op == 'Const':\n",
    "            tensor = n.attr['value'].tensor\n",
    "            size = len(tensor.tensor_content)\n",
    "            if size > max_const_size:\n",
    "                tensor.tensor_content = \"<stripped %d bytes>\"%size\n",
    "    return strip_def\n",
    "\n",
    "def show_graph(graph_def, max_const_size=32):\n",
    "    \"\"\"Visualize TensorFlow graph.\"\"\"\n",
    "    if hasattr(graph_def, 'as_graph_def'):\n",
    "        graph_def = graph_def.as_graph_def()\n",
    "    strip_def = strip_consts(graph_def, max_const_size=max_const_size)\n",
    "    code = \"\"\"\n",
    "        <script>\n",
    "          function load() {{\n",
    "            document.getElementById(\"{id}\").pbtxt = {data};\n",
    "          }}\n",
    "        </script>\n",
    "        <link rel=\"import\" href=\"https://tensorboard.appspot.com/tf-graph-basic.build.html\" onload=load()>\n",
    "        <div style=\"height:600px\">\n",
    "          <tf-graph-basic id=\"{id}\"></tf-graph-basic>\n",
    "        </div>\n",
    "    \"\"\".format(data=repr(str(strip_def)), id='graph'+str(np.random.rand()))\n",
    "\n",
    "    iframe = \"\"\"\n",
    "        <iframe seamless style=\"width:1200px;height:620px;border:0\" srcdoc=\"{}\"></iframe>\n",
    "    \"\"\".format(code.replace('\"', '&quot;'))\n",
    "    display(HTML(iframe))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe seamless style=\"width:1200px;height:620px;border:0\" srcdoc=\"\n",
       "        <script>\n",
       "          function load() {\n",
       "            document.getElementById(&quot;graph0.378627384583&quot;).pbtxt = 'node {\\n  name: &quot;Const&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n          dim {\\n            size: 3\\n          }\\n        }\\n        tensor_content: &quot;\\\\r\\\\000\\\\000\\\\000\\\\026\\\\000\\\\000\\\\000S\\\\000\\\\000\\\\000&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Const_1&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n        }\\n        int_val: 7\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;Const_1&quot;\\n  input: &quot;Const&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Const_2&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n          dim {\\n            size: 1\\n          }\\n        }\\n        int_val: 0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;Mul&quot;\\n  input: &quot;Const_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;Tidx&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;strided_slice/stack&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n          dim {\\n            size: 1\\n          }\\n        }\\n        int_val: 0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;strided_slice/stack_1&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n          dim {\\n            size: 1\\n          }\\n        }\\n        int_val: 1\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;strided_slice/stack_2&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n          dim {\\n            size: 1\\n          }\\n        }\\n        int_val: 1\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;strided_slice&quot;\\n  op: &quot;StridedSlice&quot;\\n  input: &quot;Const&quot;\\n  input: &quot;strided_slice/stack&quot;\\n  input: &quot;strided_slice/stack_1&quot;\\n  input: &quot;strided_slice/stack_2&quot;\\n  attr {\\n    key: &quot;Index&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;begin_mask&quot;\\n    value {\\n      i: 0\\n    }\\n  }\\n  attr {\\n    key: &quot;ellipsis_mask&quot;\\n    value {\\n      i: 0\\n    }\\n  }\\n  attr {\\n    key: &quot;end_mask&quot;\\n    value {\\n      i: 0\\n    }\\n  }\\n  attr {\\n    key: &quot;new_axis_mask&quot;\\n    value {\\n      i: 0\\n    }\\n  }\\n  attr {\\n    key: &quot;shrink_axis_mask&quot;\\n    value {\\n      i: 1\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;add&quot;\\n  op: &quot;Add&quot;\\n  input: &quot;Sum&quot;\\n  input: &quot;strided_slice&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\n';\n",
       "          }\n",
       "        </script>\n",
       "        <link rel=&quot;import&quot; href=&quot;https://tensorboard.appspot.com/tf-graph-basic.build.html&quot; onload=load()>\n",
       "        <div style=&quot;height:600px&quot;>\n",
       "          <tf-graph-basic id=&quot;graph0.378627384583&quot;></tf-graph-basic>\n",
       "        </div>\n",
       "    \"></iframe>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# (Load code from previous cell -- make sure to have executed above cell first.)\n",
    "%run -i _derived/2_visualize_graph.py\n",
    "\n",
    "show_graph(tf.get_default_graph())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 The Session\n",
    "\n",
    "So far we have only setup our computational Graph -- If you want to actually\n",
    "*do* any computations, you need to attach the graph to a Session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The only difference to a \"normal\" session is that the interactive\n",
    "# session registers itself as default so .eval() and .run() methods\n",
    "# know which session to use...\n",
    "interactive_session = tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "826\n"
     ]
    }
   ],
   "source": [
    "# Hooray -- try printing other tensors of above to see the intermediate\n",
    "# steps. What is their type and shape ?\n",
    "print favorite_sum.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\n"
     ]
    }
   ],
   "source": [
    "# Note that the session is also connected to a Graph, and if no Graph\n",
    "# is specified then it will connect to the default Graph. Try to fix\n",
    "# the following code snippet:\n",
    "\n",
    "graph2 = tf.Graph()\n",
    "with graph2.as_default():\n",
    "    graph2_tensor = tf.constant([1])\n",
    "with tf.Session(graph=graph2) as sess: #with tf.Session() as sess:\n",
    "    print graph2_tensor.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n",
      "123456\n"
     ]
    }
   ],
   "source": [
    "# Providing input to the graph: The value of any tensor can be overwritten\n",
    "# by the \"feed_dict\" parameter provided to Session's run() method:\n",
    "a = tf.constant(1)\n",
    "b = tf.constant(2)\n",
    "a_plus_b = tf.add(a, b)\n",
    "print interactive_session.run(a_plus_b)\n",
    "print interactive_session.run(a_plus_b, feed_dict={a: 123000, b:456})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "42\n"
     ]
    }
   ],
   "source": [
    "# It's good practice not to override just any tensor in the graph, but to\n",
    "# rather use \"tf.placeholder\" that indicates that this tensor must be\n",
    "# provided through the feed_dict:\n",
    "placeholder = tf.placeholder(tf.int32)\n",
    "placeholder_double = 2 * placeholder\n",
    "### YOUR ACTION REQUIRED:\n",
    "# Modify below command to make it work.\n",
    "print placeholder_double.eval(feed_dict={placeholder:21}) #print placeholder_double.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 The Shapes\n",
    "\n",
    "Another basic skill with Tensorflow is the handling of shapes. This\n",
    "sounds pretty simple but you will be surprised by how much time of\n",
    "your Tensorflow coding you will spend on massaging Tensors in the\n",
    "right form...\n",
    "\n",
    "Here we go with a couple of exercises with increasing difficulty...\n",
    "\n",
    "Please refer to the Tensorflow documentation\n",
    "[Tensor Transformations](https://www.tensorflow.org/versions/master/api_guides/python/array_ops#Shapes_and_Shaping)\n",
    "for useful functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"Const_5:0\", shape=(12,), dtype=int32)\n",
      "Tensor(\"Placeholder_1:0\", shape=(?, 3), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "tensor12 = tf.constant([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])\n",
    "print tensor12\n",
    "batch = tf.placeholder(tf.int32, shape=[None, 3])\n",
    "print batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'mul_1:0' shape=(12,) dtype=float32>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Tensor must be of same datatype. Try to change the datatype\n",
    "# of one of the tensors to fix the ValueError...\n",
    "multiplier = tf.constant(1.5)\n",
    "### YOUR ACTION REQUIRED:\n",
    "# Fix error below.\n",
    "tf.cast(tensor12, tf.float32) * multiplier #tensor12 * multiplier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3, 2, 2) (3, 2, 2, 1)\n"
     ]
    }
   ],
   "source": [
    "# What does tf.squeeze() do? Try it out on tensor12_3!\n",
    "tensor12_3 = tf.reshape(tensor12, [3, 2, 2, 1])\n",
    "### YOUR ACTION REQUIRED:\n",
    "# Checkout the effects of tf.squeeze()\n",
    "print tf.squeeze(tensor12_3).shape, tensor12_3.shape #print tensor12_3.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first elements:\n",
      "[1 2 3]\n",
      "[[1]\n",
      " [2]\n",
      " [3]]\n"
     ]
    }
   ],
   "source": [
    "# This cell is about accessing individual elements of a 2D tensor:\n",
    "batch = tf.constant([[1, 2, 3, 0, 0],\n",
    "                     [2, 4, 6, 8, 0],\n",
    "                     [3, 6, 0, 0, 0]])\n",
    "# Note that individual elements have lengths < batch.shape[1] but\n",
    "# are zero padded.\n",
    "lengths = tf.constant([3, 4, 2])\n",
    "\n",
    "# The FIRST elements can be accessed by using Python's\n",
    "# overloaded bracket indexing OR the related tf.slice():\n",
    "print 'first elements:'\n",
    "print batch[:, 0].eval()\n",
    "print tf.slice(batch, [0, 0], [3, 1]).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "last elements:\n",
      "[3 8 6]\n"
     ]
    }
   ],
   "source": [
    "# Accessing the LAST (non-padded) element within every sequence is\n",
    "# somewhat more involved -- you need to specify both the indices in\n",
    "# the first and the second dimension and then use tf.gather_nd():\n",
    "### YOUR ACTION REQUIRED:\n",
    "# Define provide the correct expression for indices_0 and indices_1.\n",
    "indices_0 = range(3) #indices_0 =\n",
    "indices_1 = lengths - 1 #indices_1 =\n",
    "print 'last elements:'\n",
    "print tf.gather_nd(batch, tf.transpose([indices_0, indices_1])).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1]\n",
      " [3]\n",
      " [5]\n",
      " [7]\n",
      " [9]]\n"
     ]
    }
   ],
   "source": [
    "# Below you have an integer tensor and then an expression that is set True\n",
    "# for all elements that are odd. Try to print those elements using the\n",
    "# operations tf.where() and tf.gather()\n",
    "numbers = tf.range(1, 11)\n",
    "odd_condition = tf.logical_not(tf.equal(0, tf.mod(numbers, 2)))\n",
    "### YOUR ACTION REQUIRED:\n",
    "# Define provide the correct expression for odd_indices and odd_numbers.\n",
    "odd_indices = tf.where(odd_condition) #odd_indices =\n",
    "odd_numbers = tf.gather(numbers, odd_indices) #odd_numbers =\n",
    "print odd_numbers.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"Placeholder_2:0\", shape=(?, 2), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "# \"Dynamic shapes\" : This feature is mainly used for variable size batches.\n",
    "# \"Dynamic\" means that one (or multiple) dimensions are not specified\n",
    "# before graph execution time (when running the graph with a session).\n",
    "batch_of_pairs = tf.placeholder(dtype=tf.int32, shape=(None, 2))\n",
    "\n",
    "# Note how the \"unknown\" dimension displays as a \"?\".\n",
    "print batch_of_pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2 3 4]\n",
      " [5 6 7 8]]\n"
     ]
    }
   ],
   "source": [
    "# So we want to reshape the batch of pairs into a batch of quadruples.\n",
    "# Since we don't know the batch size at runtime we will use the special\n",
    "# value \"-1\" (meaning \"as many as needed\") for the first dimension.\n",
    "# (Note that this wouldn't work for batch_of_triplets.)\n",
    "### YOUR ACTION REQUIRED:\n",
    "# Complete next line.\n",
    "batch_of_quadruples = tf.reshape(batch_of_pairs, [-1, 4]) #batch_of_quadruples = tf.reshape(batch_of_pairs, \n",
    "\n",
    "# Test run our batch of quadruples:\n",
    "print batch_of_quadruples.eval(feed_dict={\n",
    "    batch_of_pairs: [[1,2], [3,4], [5,6], [7,8]]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([Dimension(None), Dimension(2)])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Dynamic shapes cannot be accessed at graph construction time;\n",
    "# accessing the \".shape\" attribute (which is equivalent to the\n",
    "# .get_shape() method) will return a \"TensorShape\" with \"Dimension(None)\".\n",
    "batch_of_pairs.shape\n",
    "\n",
    "# i.e. .shape is a property of every tensor that can contain\n",
    "# values that are not specified -- Dimension(None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# i.e. first dimension is dynamic and only known at runtime\n",
    "batch_of_pairs.shape[0].value == None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2], dtype=int32)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The actual dimensions can only be determined at runtime\n",
    "# by calling tf.shape() -- the output of the tf.shape() Op\n",
    "# is a tensor like any other tensor whose value is only known\n",
    "# at runtime (when also all dynamic shapes are known).\n",
    "batch_of_pairs_shape = tf.shape(batch_of_pairs)\n",
    "batch_of_pairs_shape.eval(feed_dict={\n",
    "    batch_of_pairs: [[1, 2]]\n",
    "})\n",
    "\n",
    "# i.e. tf.shape() is an Op that takes a tensor (that might have\n",
    "# a dynamic shape or not) as input and outputs another tensor\n",
    "# that fully specifies the shape of the input tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2, 2],\n",
       "       [3, 3],\n",
       "       [4, 4]], dtype=int32)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# So you think shapes are easy, right?\n",
    "# Well... Then here we go with a real-world shape challenge!\n",
    "#\n",
    "# (You probably won't have time to finish this challenge during\n",
    "# the workshop; come back to this later and don't feel bad about\n",
    "# consulting the solution...)\n",
    "#\n",
    "# Imagine you have a recurrent neural network that outputs a \"sequence\"\n",
    "# tensor with dimension [?, max_len, ?], where\n",
    "# - the first (dynamic) dimension is the number of elements in the batch\n",
    "# - the second dimension is the maximum sequence length\n",
    "# - the third (dynamic) dimension is the number of number per element\n",
    "#\n",
    "# The actual length of every sequence in the batch (<= max_len) is also\n",
    "# specified in the tensor \"lens\" (length=number of elements in batch).\n",
    "#\n",
    "# The task at hand is to extract the \"nth\" element of every sequence.\n",
    "# The resulting tensor \"last_elements\" should have the shape [?, ?],\n",
    "# matching the first and third dimension of tensor \"sequence\".\n",
    "#\n",
    "# Hint: The idea is to reshape the \"sequence\" to \"partially_flattened\"\n",
    "# and then construct a \"idxs\" tensor (within this partially flattened\n",
    "# tensor) that returns the requested elements.\n",
    "#\n",
    "# Handy functions:\n",
    "# tf.gather()\n",
    "# tf.range()\n",
    "# tf.reshape()\n",
    "# tf.shape()\n",
    "\n",
    "lens = tf.placeholder(dtype=tf.int32, shape=(None,))\n",
    "max_len = 5\n",
    "sequences = tf.placeholder(dtype=tf.int32, shape=(None, max_len, None))\n",
    "### YOUR ACTION REQUIRED:\n",
    "# Find the correct expression for below tensors.\n",
    "batch_size = tf.shape(sequences)[0] #batch_size = \n",
    "hidden_state_size = tf.shape(sequences)[2] #hidden_state_size = \n",
    "idxs = tf.range(0, batch_size) * max_len + (lens - 1) #idxs = \n",
    "partially_flattened = tf.reshape(sequences, [-1, hidden_state_size]) #partially_flattened =\n",
    "last_elements = tf.gather(partially_flattened, idxs) #last_elements =\n",
    "\n",
    "sequences_data = [\n",
    "    [[1,1], [1,1], [2,2], [0,0], [0,0]],\n",
    "    [[1,1], [1,1], [1,1], [3,3], [0,0]],\n",
    "    [[1,1], [1,1], [1,1], [1,1], [4,4]],\n",
    "]\n",
    "lens_data = [3, 4, 5]\n",
    "# Should output [[2,2], [3,3], [4,4]]\n",
    "last_elements.eval(feed_dict={sequences: sequences_data, lens: lens_data})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4 Variables – bonus!\n",
    "\n",
    "So far all our computations have been purely stateless. Obviously,\n",
    "programming become much more fun once we add some state to our code...\n",
    "Tensorflow's **variables** encode state that persists between calls to\n",
    "`Session.run()`.\n",
    "\n",
    "The confusion with Tensorflow and variables comes from the fact that we\n",
    "usually \"execute\" the graph from within Python by running some nodes of\n",
    "the graph -- via `Session.run()` -- and that variable assignments are also\n",
    "encoded through nodes in the graph that only get executed if we ask the\n",
    "value of one of its descendants (see explanatory code below).\n",
    "\n",
    "Tensorflow's overview of\n",
    "[variable related functions](https://www.tensorflow.org/versions/r1.0/api_guides/python/state_ops#Variables),\n",
    "the\n",
    "[variable HOWTO](https://www.tensorflow.org/versions/r1.0/programmers_guide/variables),\n",
    "and the\n",
    "[variable guide](https://www.tensorflow.org/programmers_guide/variables).\n",
    "\n",
    "And finally some notes on [sharing variables](https://www.tensorflow.org/api_guides/python/state_ops#Sharing_Variables)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "counter = tf.Variable(0)\n",
    "increment_counter = tf.assign_add(counter, 1)\n",
    "with tf.Session() as sess:\n",
    "    # Something is missing here...\n",
    "    # -> Search the world wide web for the error message...\n",
    "    ### YOUR ACTION REQUIRED:\n",
    "    # Add a statement that fixes the error.\n",
    "    sess.run([tf.global_variables_initializer()]) #\n",
    "    print increment_counter.eval()\n",
    "    print increment_counter.eval()\n",
    "    print increment_counter.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "# Same conditions apply when we use our global interactive session...\n",
    "interactive_session.run([tf.global_variables_initializer()])\n",
    "print increment_counter.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n"
     ]
    }
   ],
   "source": [
    "# Execute this cell multiple times and note how our global interactive\n",
    "# sessions keeps state between cell executions.\n",
    "print increment_counter.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n"
     ]
    }
   ],
   "source": [
    "# Usually you would create variables with tf.get_variable() which makes\n",
    "# it possible to \"look up\" variables later on.\n",
    "\n",
    "# For a change let's not try to fix a code snippet but rather to make it\n",
    "# fail:\n",
    "# 1. What happens if the block is not wrapped in a tf.Graph()?\n",
    "# 2. What happens if reuse= is not set?\n",
    "# 3. What happens if dtype= is not set?\n",
    "\n",
    "with tf.Graph().as_default():\n",
    "    with tf.variable_scope('counters'):\n",
    "        counter1 = tf.get_variable('counter1', initializer=1)\n",
    "        counter2 = tf.get_variable('counter2', initializer=2)\n",
    "        counter3 = tf.get_variable('counter3', initializer=3)\n",
    "    with tf.Session() as sess:\n",
    "        sess.run([tf.global_variables_initializer()])\n",
    "        print counter1.eval()\n",
    "        with tf.variable_scope('counters', reuse=True):\n",
    "            print tf.get_variable('counter2', dtype=tf.int32).eval()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
